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FOREWORD 


This  research  and  development  was  performed  in  response  to  recognized  Navy  needs 
for  an  investigation  of  the  effects  of  initial  training  and  job  conditions  on  skill  retention 
under  subproject  RF63-522-001  (Computer-aided  and  Classroom  Training),  work  unit  03.07 
(Skill  and  Knowledge  Retention).  This  issue  was  initially  addressed  in  a  paper  prepared  in 
1976  in  response  to  a  request  from  the  Chief  of  Naval  Operations  (OP-96),  citing  the 
concern  that  training  and  job  conditions  might  not  be  conducive  to  the  maintenance  of 
competent  performance.  The  objectives  of  the  subproject  are  to  (1)  derive  ways  of 
detecting  potential  problems  existing  in  the  Navy  and  (2)  recommend  means  to  minimize 
performance  deterioration  by  restructuring  training  and  job  conditions. 

This  is  the  sixth  report  concerned  with  skill  deterioration.  Previous  reports  described 
long-term  retention  of  factual  information  in  the  Propulsion  Engineering  Basics  Course, 
the  effects  of  different  instructional  strategies  on  long-term  retention  of  materials  taken 
from  that  course,  existing  research  relevant  to  skill  and  knowledge  loss,  a  survey  of  the 
three  Navy  sonar  communities --aviator,  subsurface,  and  surface — to  identify  conditions 
leading  to  skill  loss,  and  an  assessment  of  job  and  training  variables  leading  to  skill  loss  in 
sonar  technicians  (surface)  (STGs)  (NPRDC  TN  80-5,  TR  81-22,  SR  82-21,  SR  83-18,  and 
SR  83-26).  This  report  describes  an  assessment  of  those  variables  for  aviation 
antisubmarine  warfare  operators  (AWs).  Assessments  were  conducted  using  a  job 
condition  questionnaire. 

The  empirical  results  of  this  work  are  primarily  intended  for  the  Commander, 
Antisubmarine  Warfare  Wing,  Pacific  (COMASWWINGPAC),  and  for  other  agencies  con¬ 
cerned  with  AW  training.  The  development  of  the  questionnaire  method  of  performance 
assessment  that  uses  the  quality  of  initial  training  and  job  conditions  to  estimate  the 
likelihood  of  skill  degradation  should  be  of  interest  to  personnel  and  training  managers. 

Appreciation  is  expressed  to  the  staff  at  the  ASW  Operational  Center,  Naval  Air 
Station,  North  Island,  particularly  AWCS  Guyette,  and  to  the  staff  of  the  Training 
Department  at  COMASWWINGPAC,  particularly  AWCM  Helenihi,  AWCM  Sweeney,  and 
AWC  Mauffrey,  for  their  time,  expertise,  and  aid  in  this  research  effort. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  W.  TWEEDDALE 
Technical  Director 


SUMMARY 


Problem 


Navy  personnel  in  technical  ratings  must  maintain  high  levels  of  skill  and  knowledge 
to  be  able  to  perform  their  jobs  correctly  when  required.  However,  job  conditions  may 
not  support  maintaining  critical  skills  at  the  required  levels  because  of  infrequent 
practice  of  tasks,  inadequate  feedback  about  the  quality  of  the  performance,  long  periods 
of  nonutilization  of  specific  skills,  and  interference  due,  for  example,  to  assignment  to 
task-irrelevant  duties.  Since  the  cost  of  measuring  performance  to  monitor  skill 
degradation  directly  is  prohibitive,  an  indirect  method  for  predicting  skill  loss  is  needed. 

Objective 

The  objective  of  this  work  was  to  identify  job  conditions  associated  with  skill  and 
knowledge  degradation  in  aviation  antisubmarine  warfare  operators  (AWs)  using  a 
specially  developed  questionnaire. 

Approach 


A  sample  of  42  AWs,  with  varying  levels  of  training  and  experience  in  operating  sonar 
systems,  was  administered  an  experimental  survey  questionnaire  designed  to  provide 
information  about  initial  training  and  job  conditions  associated  with  performing  acoustic 
analysis.  In  addition,  direct  performance  measures  for  both  acoustic  ASW  knowledge 
factors  and  acoustic  analysis/classification  on  the  lofargram  were  obtained. 

Findings 

Findings  support  those  of  earlier  efforts  showing  that  the  AW  community  is 
characterized  by  mastery  level  initial  training,  relatively  frequent  task  practice, 
individual  feedback,  and  short  periods  of  task  nonutilization.  Subjects  performed  well  on 
the  direct  performance  tests  that  measured  knowledge  and  passive  acoustic  analysis  skills. 

Conclusions 


Results  indicate  that  the  questionnaire  method  accurately  predicts  good  performance 
as  well  as  bad  performance.  The  general  finding  of  this  work  indicates  that  good  initial 
training  and  job  conditions  are  related  to  good  performance  on  objective  tests.  It  is 
probable  that  this  general  technique  can  be  used  to  assess  the  likelihood  of  poor 
performance  due  to  skill  degradation  and/or  training  inadequacies  in  other  technical 
ratings.  Further,  the  categories  of  conditions  may  be  used  to  suggest  corrective  action. 

Recommendation 


Job  condition  questionnaires  should  be  developed  for  other  technical  ratings  to 
determine  if  this  method  is  generalizable  to  other  technical  tasks. 
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INTRODUCTION 


Problem  and  Background 


The  modern  Navy  is  a  complex,  highly  technical  organization.  Approximately  75 
percent  of  Navy  ratings  require  technically  skilled  personnel  who  must  most  often 
perform  procedural  tasks  (i.e.,  those  requiring  the  operator  to  perform  multiple  steps  in 
the  correct  sequence  to  accomplish  the  job).  Such  tasks  are  difficult  to  learn,  easy  to 
forget,  and  the  most  likely  to  result  in  critical  errors.  Vineberg  and  Joyner  (1982),  in  a 
review  of  military  studies,  found  that  the  ratings  classified  as  hardest  to  learn  and 
requiring  the  most  training  were  those  in  which  it  was  most  often  predicted  that  errors 
would  result  in  failure  to  carry  out  missions  or  in  losses  of  ships,  planes,  and  personnel. 


It  has  been  demonstrated  that  the  level  of  initial  training  and  the  quality  of  job 
conditions  affect  the  degree  to  which  procedural  skills  are  degraded  over  time  (Hurlock  6c 
Montague,  1982).  Since  investigation  has  shown  that  ratings  differ  as  to  the  quality  of 
initial  training  and  job  conditions  (Wetzel  6c  Montague,  1983),  some  ratings  may  be  more 
vulnerable  to  skills  loss  than  others. 


Navy  personnel  in  technical  ratings  must  be  well  trained  and  must  maintain  high 
levels  of  skill  and  knowledge  to  be  able  to  perform  their  jobs  correctly  when  required. 
Yet  training  for  particular  skills  may  suffer  from  lack  of  resources  or  time  because  of  the 
need  to  economize  or  because  of  other  tradeoffs.  Furthermore,  to  maintain  skill  on 
complex  procedural  tasks,  individuals  need  frequent  opportunity  to  practice  that  skill  and 
receive  feedback  about  the  quality  of  their  performance  on  the  task.  Failure  to  provide 
opportunity  to  practice  with  adequate  feedback  is  likely  to  produce  skill  degradation.  The 
degradation  of  these  critical  skills  directly  affects  personnel  readiness  and  is  of  real 
concern  to  the  Navy. 

In  an  earlier  study  (Wetzel  6c  Montague,  1982),  subject  matter  experts  (SMEs)  in  the 
Navy's  three  sonar  communities — surface,  subsurface,  and  aviation — were  surveyed  to 
identify  training  and  job  conditions  indicative  of  serious  skill  degradation  problems. 
Analysis  of  interview  results  showed  that,  because  of  the  training  and  job  conditions 
experienced  by  surface  sonar  technicians  (STGs),  it  was  highly  probable  that  their  skill 
would  degrade  over  time.  Thus,  in  a  follow-on  effort,  a  sample  of  STGs  was  administered 
a  questionnaire  requiring  subjective  estimates  of  initial  training  and  job  conditions 
(Wetzel,  Konoske,  6c  Montague,  1983).  Analysis  of  questionnaire  responses  supported  the 
conclusions  reached  by  Wetzel  and  Montague;  that  is,  they  suggested  that  training  quality 
was  marginal,  task  nonutilization  periods  after  initial  training  were  long,  on-job  practice 
was  infrequent,  and  feedback  about  task  performance  was  inadequate.  When  direct 
measures  of  performance  were  used  to  assess  STG  competency  on  critical  sonar  skills, 
their  test  scores  were  very  low.  The  results  of  this  research  effort  indicated  that  the 
questionnaire  method  was  useful  in  identifying  a  potential  performance  problem  in  STGs. 

For  the  aviation  sonar  community,  Wetzel  and  Montague  found  that  the  initial  level 
of  learning  was  high,  practice  of  the  critical  job  tasks  was  fairly  frequent,  individual 
feedback  about  performance  was  consistent  and  of  good  quality,  and  periods  of  task 
nonutilizaiton  were  short.  Because  of  the  relatively  high  quality  of  the  training  and  job 
conditions,  it  was  expected  that  skill  levels  on  these  critical  tasks  would  be  comparably 


‘Fredericks,  P.  and  Montague,  W.  E.  San  Diego:  Navy  Personnel  Research  and 
Development  Center,  Letter  report  to  Chief  of  Naval  Education  and  Training,  Ser  638,  16 
October  1980;  subj:  IQI  evaluation  of  fleet  feedback  questionnaire. 


high  when  measured.  However,  the  cost  of  monitoring  skill  levels  by  direct  performance 
measurement  is  prohibitive.  It  requires  considerable  time  from  testees  as  well  as  from 
instructors  or  supervisors,  and  it  ties  up  equipment.  It  also  requires  the  development  of  a 
standardized  system  for  carrying  out  the  testing,  scoring,  data  storage,  and  reporting. 
The  usefulness  of  an  alternative  method  of  proficiency  assessment  that  uses  the  quality  of 
initial  training  and  job  conditions  to  estimate  the  likelihood  of  skill  degradation  needs  to 
be  evaluated. 

Objective 

The  objective  of  this  work  was  to  identify  initial  training  and  job  conditions 
associated  with  skill  and  knowledge  degradation  for  aviation  antisubmarine  warfare 
operators  (AWs).  It  was  undertaken  to  evaluate  further  the  questionnaire  method  for 
assessing  the  expected  performance  quality  of  AW  operators.  Since  results  of  the  prior 
study  (Wetzel  &  Montague,  1983)  suggested  good  conditions  of  training  and  skill 
maintenance  for  AW's,  it  was  expected  that  those  results  would  be  reconfirmed. 


APPROACH 


Subjects  and  Procedure 

Forty-two  AWs,  ranging  in  rank  from  chief  petty  officer  to  third  class  petty  officer, 
were  selected  on  the  basis  of  their  participation  in  the  ASWWINGPAC  Quarterly  Exam. 
All  subjects  were  members  of  aviation  ASW  squadrons  (S3-A)  stationed  at  the  Naval  Air 
Station,  North  Island,  and  all  had  training  plus  varying  amounts  of  experience  in 
performing  acoustic  analysis. 

A  job  condition  questionnaire  was  administered  to  the  subjects  immediately  prior  to 
the  administration  of  the  ASW  Wing  quarterly  examination,  which  is  administered  on  a 
regularly  scheduled  basis  to  all  acoustic  analysts  stationed  at  NAS  North  Island.  All  of 
the  subjects  were  tested  in  one  test  administration.  The  examination  was  scored  by  the 
experimenters  using  grading  criteria  supplied  by  the  ASW  Wing  staff. 

Materials 

3ob  Condition  Questionnaire 

The  job  condition  questionnaire  was  an  experimental  survey  device  designed  to  assess 
the  quality  of  initial  training  in  acoustic  analysis,  practice  on  the  job,  feedback 
conditions,  and  lengths  of  nonutilization  periods.  Questionnaire  items  are  shown  in  Figure 
1. 


Performance  Measure 


The  purpose  of  the  ASW  Wing  quarterly  examination  is  to  assess  operator  proficiency 
of  both  knowledge  factors  and  lofargram  analysis  and  classification  skills.  The  knowledge 
factors  are  tested  on  a  paper  and  pencil  test,  consisting  of  50  multiple-choice  and  fill-in 
questions  that  cover  physics  of  sound,  acoustic  intelligence,  and  computation  items.  The 
lofargram  analysis  and  classification  test  requires  the  operator  to  perform  analysis  and 
classification  procedures  on  five  static,  linear  lofargrams. 
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JOB  CONDITION  QUESTIONNAIRE 


Formal  Training 

1.  How  long  ago  was  your  last  formal  training  (FRS)  in  gram  analysis?  (Circle  the 
number  of  months) 

1  2  3  4  5  6  7  8  9  10  II  12  13  14  15  longer 

2.  How  well  could  you  perform  acoustic  analysis/classification  after  you  had  completed 
FRS  training?  (Circle  one) 

abed  (where  a  =  never  could  perform  the  task  correctly,  b  -  could  perform  the 
task  correctly  half  of  the  time,  c  =  could  perform  the  task  correctly  most  of  the 
time,  and  d  =  could  perform  the  task  correctly  every  time) 

3.  While  you  were  in  FRS,  which  of  the  following  best  describes  your  ability  to  do  the 
task?  (Circle  one) 

abed  (where  a  =  skills  in  analysis  and  classification  were  equally  poor,  b  =  could 
do  both  very  well  but  better  at  analysis,  c  =  could  do  both  very  well  but  better  at 
classification,  and  d  -  skills  in  analysis  and  classification  were  equally  good) 

Practice 


4.  How  many  deployments  (i.e.,  6-month  cruise)  have  you  made  as  an  acoustic  analyst? 
(Circle  one) 

a  b  c  d  e  (where  a  =  0,  b  =  1,  c  -  2,  d  -=  3,  and  e  =  4  or  more) 

5.  How  many  hours  per  week  do  you  average  actually  performing  acoustic  ASW  in 
flight?  (Circle  one) 

a  b  c  d  e  (where  a  r  0-1  hours,  b  t  2-3  hours,  c  =  4-6  hours,  d  r  7-10  hours,  and 
e  =  more  than  10  hours) 

6.  How  many  hours  per  week  do  you  average  studying  or  practicing  acoustic  ASW? 
(Circle  one) 

a  b  c  d  e  (where  a  =  0-1  hours,  b  -  2-3  hours,  c  r  4-6  hours,  d  =  7-10  hours,  and 
e  =  more  than  10  hours) 

7.  Since  the  last  wing  test  (December),  what  is  the  longest  period  of  time  that  you  have 
not  performed  analysis  and  classification?  (Circle  the  number  of  weeks) 

1  2  3  4  5  6  7  8  9  10  11  12 

Feedback  (In  answering  the  following  questions,  circle  one) 

a  b  c  d  e  (where  a  -  never,  b  =  rarely,  c  -  some  of  the  time,  d  -  most  of  the  time, 
and  e  =  always) 

8.  When  you  have  correctly  classified  a  target  on  an  exercise,  do  you  receive  feedback 
about  your  analysis  (sound  source,  predominant  spectrum,  line  characteristics)? 

9.  When  you  have  correctly  classified  a  target  on  an  exercise,  do  you  receive  feedback 
about  intelligence  information  (ERPM  range,  TPK)? 

10.  When  you  have  incorrectly  classified  a  target  on  an  exercise,  do  you  receive  feedback 
about  your  analysis  of  the  target'’ 

1 1.  When  you  have  incorrectly  classified  a  target  on  an  exercise,  do  you  receive  feedback 
about  intelligence  information’’ 


Figure  1.  Job  condition  questionnaire. 


FINDINGS 


Correlational  analysis  between  results  of  the  job  condition  questionnaire  and  the 
performance  test  scores  was  precluded  because  of  low  variability  among  responses  to  the 
job  condition  questionnaire  and  the  test  scores. 

Job  Condition  Questionnaire 

Level  of  Initial  Learning 


Table  1,  which  provides  frequencies  for  operator  estimates  of  the  degree  of  original 
learning  on  acoustic  analysis  and  classification  procedures  (items  2  and  3),  shows  that 
their  estimates  were  relatively  high.  Sixty-nine  percent  of  the  subjects  estimated  that 
they  could  perform  acoustic  analysis  correctly  most  of  the  time  after  training;  and  26 
percent,  that  they  could  perform  the  tasks  correctly  all  of  the  time.  Only  one  student 
estimated  that  his  skills  in  acoustic  analysis  and  classification  procedures  were  poor  while 
in  training.  Approximately  26  percent  of  the  subjects  reported  having  had  formal  training 
in  acoustic  analyses  and  classification  within  the  previous  6  months  (item  1). 

Table  1 

Subjects'  Estimates  of  Acoustic  Analysis  and 
Classification  Skills  Over  Time 


Response 

Frequency 

Percent3 

Description  of  Skills  While  in  Training  at  FRS  (Item  3) 

Skills  in  analysis  and  classification  were  equally 
poor. 

1 

2.4 

Could  do  both  very  well  but  better  at  analysis. 

1 

2.4 

Could  do  both  very  well  but  better  at  classification. 

9 

21.4 

Skills  in  analysis  and  classification  were  equally  good. 

30 

71.4 

No  response 

1 

2.4 

Description  of  Skills  After  Training  (Item  2) 


Never  could  perform  task  correctly. 

0 

0.0 

Could  perform  task  correctly  half  of  the  time. 

2 

4.8 

Could  perform  task  correctly  most  of  the  time. 

29 

69.0 

Could  perform  task  correctly  every  time. 

11 

26.2 

Number  of  Months  Since  Last  Formal  Training  in  Acoustic 
Analysis/Classification  (Item  1) 


0-3  months  7 

4-6  months  4 

7-9  months  6 

10-12  months  7 

13  months  or  longer  17 

No  response  1 


16.7 


16.7 

40.5 

2.4 


totals  do  not  always  equal  100  because  of  rounding. 


Practice 


Table  2  provides  frequencies,  means,  and  standard  deviations  (SDs)  for  items  concern¬ 
ing  the  amount  of  practice  (items  5  and  6).  Over  85  percent  of  the  subjects  said  they 
spent  only  3  hours  a  week  or  less  practicing  acoustic  ASW  in  flight.  However,  54  percent 
said  they  practiced  or  studied  gram  analysis  for  at  least  4  hours  per  week;  and  31  percent, 
over  10  hours  per  week.  Only  10  percent  reported  that  they  had  made  more  than  one 
deployment  as  an  acoustic  analyst. 


Table  2 

Subjects'  Estimates  of  Amount  of  Practice 


Response 

Frequency 

1  cent3 

Time  Spent  Performing  Acoustic  ASW  in  Flight  (Item  5) 

0-3  hours  per  week 

36 

-  7 

4-6  hours  per  week 

6 

i 

7-10  hours  per  week 

0 

J.O 

More  than  10  hours  per  week 

0 

0.0 

Mean 

3.29 

SD 

0.71 

Time  Spent  Studying  or  Practicing  Acoustic  ASW  (Item  6) 

0-3  hours  per  week 

10 

23.8 

4-6  hours  per  week 

13 

30.9 

7-10  hours  per  week 

6 

14.3 

More  than  10  hours  per  week 

13 

30.9 

Mean 

7.26 

SD 

3.68 

Time  Spent  Deployed  as  Acoustic  Analyst  (Item  4) 

0  deployments 

17 

40.5 

1  deployment 

21 

50.0 

2  deployments 

1 

2.4 

3  deployments 

I 

2.4 

4  or  more  deployments 

2 

4.8 

totals  do  not  always  equal  100  because  of  rounding. 


Feedback 

Table  3  presents  frequencies  for  the  amount  and  type  of  feedback  received.  Sixty- 
two  percent  of  the  subjects  reported  that  they  receive  feedback  on  analysis  procedures 
most  or  all  of  the  time  when  they  have  incorrectly  classified  a  target,  compared  to  45 
percent  when  they  have  classified  the  target  correctly.  For  acoustic  intelligence,  38 
percent  reported  receiving  feedback  most  or  all  of  the  time  when  they  classified  a  target 
incorrectly;  and  29  percent,  when  they  classified  a  target  correctly. 


Table  3 

Subjects'  Estimates  of  Frequency  and  Type  of  Feedback 


Analysis  Intelligence 

Frequency  of  Feedback  Frequency  Percent3  Frequency  Percent3 


Correct  Classification  (Items  8  and  9) 


Never 

4 

9.5 

6 

14.3 

Rarely 

7 

16.7 

14 

33.3 

Some  of  the  time 

12 

28.6 

10 

23.8 

Most  of  the  time 

12 

28.6 

7 

16.7 

Always 

7 

16.7 

5 

11.9 

Incorrect  Classification  (Items  10  and  1 1) 

Never 

4 

9.5 

6 

14.3 

Rarely 

2 

4.8 

6 

14.3 

Some  of  the  time 

8 

19.0 

12 

28.6 

Most  of  the  time 

12 

28.6 

5 

11.9 

Always 

14 

33.3 

11 

26.2 

No  response 

2 

4.8 

2 

4.8 

totals  do  not  always  equal  100  due  to  rounding. 


Nonutilization 

Table  4,  which  provides  frequencies  for  subjects'  estimates  of  lengths  of  nonutiliza¬ 
tion  periods,  shows  that  the  periods  are  relatively  short.  About  74  percent  of  the  subjects 
have  never  had  a  period  of  more  than  4  weeks  when  they  had  not  performed  analysis  and 
classification. 

Performance  Measure 

Table  5,  which  presents  results  of  the  ASW  wing  test,  shows  that  subjects  performed 
relatively  well  overall  on  the  knowledge  test.  However,  they  performed  better  on  the 
computational  items  than  they  did  on  the  fact  items.  Also,  subjects  performed  relatively 
well  on  the  gram  test,  particularly  on  the  analysis  task. 


Table  4 


Subjects'  Estimates  o f  Periods  of  Nonutilization  (Item  7) 


Response 

Frequency 

Percent 

1-2  weeks 

20 

47.6 

3-4  weeks 

11 

26.2 

5-6  weeks 

4 

9.5 

7-8  weeks 

4 

9.5 

9-10  weeks 

0 

0.0 

11-12  weeks 

1 

2.4 

No  response 

2 

4.8 

Table  5 

Results  of  ASW  Wing  Quarterly  Exam 

Test 

Raw  Score 

Mean 

SD 

Mean  Percent 
Correct 

Knowledge  Test: 

40.03 

4.18 

80.06 

Fact  items 

30.89 

3.82 

77.22 

Computation  items 

9.13 

0.46 

91.30 

Gram  Test:a 

11.42 

1.75 

76.13 

Analysis  items 

4.42 

.33 

88.40 

Intelligence  items 

3.10 

.98 

62.00 

Classification  items 

3.83 

.46 

76.60 

N  =  23. 

aOnly  23  of  the  42  subjects  took  the  gram  test. 


DISCUSSION 

Operator  responses  to  the  job  condition  questionnaire  in  this  effort  supported  the 
structured  interview  data  obtained  from  SMEs  (Wetzel  &  Montague,  1983)  documenting  a 
high  level  of  initial  training,  relatively  frequent  practice  on  the  task,  individual 
performance  feedback  (including  information  about  errors),  and  relatively  short  periods  of 
task  nonutilization.  The  subjects  performed  at  a  high  level  of  proficiency  on  the 
knowledge  and  performance  tests,  as  predicted  by  the  questionnaire  method. 

Also,  this  study  supports  the  Wetzel  et  al.  (1983)  study,  in  that  the  questionnaire 
method  was  successful  in  assessing  the  initial  training  and  job  conditions.  It  is  probable 
that  this  general  technique  can  be  used  to  assess  the  likelihood  of  poor  performance  due 
to  skill  degradation  and/or  training  inadequacies  in  other  technical  ratings.  It  is 
important  to  note  that,  since  the  specific  items  used  relate  to  initial  training  and  job 
conditions  for  a  specific  rating,  different  questionnaires  would  have  to  be  constructed  for 


r 


each  of  the  ratings  investigated.  It  does  appear,  however,  that  the  concept  of  assessing 
level  of  initial  training,  frequency  of  practice,  amount  and  quality  of  feedback,  and 
lengths  of  task  nonutilization  is  generalizable  ^  ratings  other  than  sonar  operators. 


CONCLUSIONS 

The  questionnaire  method  of  performance  estimation  appears  to  be  useful  in 
identifying  potential  performance  problems  due  to  training  inadequacies  or  skill  degrada¬ 
tion  resulting  from  specific  job  conditions.  Further,  the  categories  of  conditions  may  be 
used  to  suggest  corrective  action. 


RECOMMENDATION 

Job  condition  questionnaires  should  be  developed  for  other  technical  ratings  to 
determine  if  this  method  is  generalizable  to  other  technical  tasks. 
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